ScaNN: Fix AVQ prefetch#1899
Conversation
rmaschal
added
improvement
| @@ -532,6 +537,8 @@ class cluster_loader { | |||
| raft::copy(res, cluster_vectors, raft::make_const_mdspan(pinned_cluster)); | |||
| raft::resource::sync_stream(res, stream_); | |||
|
|
|||
| // reset stream back to previous value | |||
| raft::resource::set_cuda_stream(res, stream); | |||
There was a problem hiding this comment.
You can create a new resources object for no cost:
| // For prefetching to overlap with other gpu work | |
| // we need to schedule copies on the provided copy stream stream_ | |
| auto copy_res = raft::resources(stream_); | |
| // htod | |
| auto h_cluster_ids = | |
| raft::make_pinned_vector_view<LabelT, int64_t>(cluster_ids_buf_.data_handle(), size); | |
| @@ -532,6 +537,8 @@ class cluster_loader { | |
| raft::copy(copy_res, cluster_vectors, raft::make_const_mdspan(pinned_cluster)); | |
| raft::resource::sync_stream(res, stream_); |
There was a problem hiding this comment.
I vote on this too. Although not very relevant here, another benefit of creating a temporary resources handle is the automatic reset of the changes if an exception is raised. However, I'd like to warn that there are caveats in doing this:
- I'd suggest to use a copy the resources handle rather than creating a new resources handle to inherit all other initialized resources from the current handle.
auto copy_res = raft::resources(res);
raft::resource::set_cuda_stream(res, stream);- Even when you do a copy: if you've used any other not-yet-initialized resources from the
copy_reshandle, it would create those resources on every invocation ofprefetch_cluster, which would incur significant overhead.
There was a problem hiding this comment.
Thank you for the comments and info. I now copy the passed resources, set the stream, and use that for scheduling copies
rmaschal
force-pushed
the
fix-avq-prefetch
branch
from
1c03c6f to
22f74e7
Compare
|
Hi @rmaschal, is this targeting release/26.04? If so, please retarget from main to release/26.04 |
|
Hi @rmaschal, looks like we are good to merge this right, or are you seeing CI failures related this PR? No need to keep merging in main, as long as the Recently Updated check is passing. You can also rerun failed jobs if they are flaky. |
|
@aamijar yes should be good, some python CI failures but they should be unrelated. |
|
Great! I will try to rebase to release/26.04. Would be nice to have in the release since it a bug fix/ perf regression fix. |
aamijar
force-pushed
the
fix-avq-prefetch
branch
from
d82cfe8 to
d5d3316
Compare
|
/ok to test d5d3316 |
|
/merge |
github-project-automation
bot
moved this from In Progress
to Done
in Unstructured Data Processing
4 participants
Switching to usage of modern RAFT in cuVS (#1837) introduced a bug where the prefetched gather for AVQ is performed using the stream associated with raft::device_resources rather than the provided stream for copying. This led to two issues:
This PR sets the stream associated with the resource to the copy stream before prefetching, and back when done.